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Abstract 

' This project revolves around studying estimators for parameters in difTerent Time Series mod- 

els and studying their assymptotic properties. We introduce various bootstrap techniques for the 
. estimators obtained. Our special emphasis is on Weighted Bootstrap. We establish the con- 

' sistency of this scheme in a AR model and its variations. Numerical calculations lend further 

f— I , support to our consistency results. Next we analyze ARCH models, and study various estimators 

■ used for different error distributions. We also present resampling techniques for estimating the 

distribution of the estimators. Finally by simulating data, we analyze the numerical properties 
, of the estimators. 

r— I 

H '. 1 Bootstrap in AR(1) model 

(-H I Let Xt be a stationary AR(1) process, that is, 

Xt = 9Xt-i + Zt for t = 1,2,... (1) 
Zt iid (0,cr2); EZf <oo; \e\<l. 

We have assumed a to be known, and 9 is the unknown parameter of interest. Then the Least Squares 
^ estimate for 9 (which is approximately the MLE in case of normal errors) is given by 

Then it can be established that 

^ . V^ik -9)^ N{0, (1 - 9')) (2) 

04 . Let us introduce two particular bootstrap techniques specially used to estimate the distribution of 

from a realization of model ([T]). 



(a) Residual Bootstrap Let Zt — Xt — 9nXt-i, t = 2,3,...,n and let Zt be the standardized 
version of Zt such that :^J2^t = and = 1- Now we draw Z^ , t ^ 1,2, . . . , N with 



X 

^ ■ 

5t 1 replacement from Zt and define 



XI - zi 

X; = 9,,XU + Z: ,t^2,...,N. 

and form the statistic 

_ t=2 

" " Et^iXtur 

Then ^ forms an estimator of 6'„ and is called the Residual Bootstrap estimator. We repeat the 
simulation process several times to estimate the distribution of 0* . 



(3) 



(b) Weighted Bootstrap Alternatively we define our resampling estimator 

(4) 



S"=2 WntXtXt-1 



Er=2^"t(^t-l)^ 

where {wnt',1 < t < n,n > 1} is sl triangular sequence of random variables, independent of {^t}- 
These are the so called "Bootstrap weights", and the estimator (U) is the Weighted Bootstrap 
Estimator. 
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1.1 A Bootstrap Central limit theorem 

Under suitable conditions on the weights to be stated below, we establish the distributional consistency 
of the Weighted Bootstrap Estimator, 0* defined in Q. To establish consistency, we will prove a 
Bootstrap CLT for which we will need the following established results: 



Result 1 (P-W theorem; see Praestgaard and Wellner(1993)) Let 

{cnj] j = 1, 2, . . . , n; ri > 1} be a triangular array of constants, and let {Unj j - 
be a triangular array of row exchangeable random variables such that as n — > cx), 



1,2,. 



. , n; ri > 1} 







3. imaxi<j<„c,2y 

4. E{Unj)=0 j = l,2, 

5. E{Ul^)^l J- = 1,2, 

7. limfc^oolini sup„^^ ^ {E{U^jI{\u„,\>k}) 



,n. n > 1 
,n. n > 1 



= 



Then under the above conditions, 



1 " 



7V(0,t2) 



(5) 



Result (HI) can be generalized by taking {c„j} random variables, independent of {Unj} and the con- 
ditions (1), (2) and (3) replaced by convergence in probability. In that case conclusion (O is replaced 
by 



P 



1 " 



{cnfj = l,...,n;n> 1} 



P[YgC]^op{1) 



(6) 



where Y - N{0,t^) and C G B{R) such that P(Y e dC) = 0. 



Result 2 Let {Xi,X2, ■ . ■ ^Xn} be the realization of the stationary AR(1) process ([7]j. Then 



< oo ya,b,k e Z+; a,b > ; k > 0. This 



kT^tZl^tZU E{X?Zl^,) whenever EZr-""-"''' 
can be established using the Martingale SLLN; see Hall and Heyde 1980. 

Let us use the notations Ps, Eb, Vb to respectively denote probabilities, expectations and variances 
with respect to the distribution of the weights, conditioned on the given data {Xi, . . . ,Xn}- The 
weights are assumed to be row exchangeable. We henceforth drop the first suffix in the weights Wni 
and denote it by Wi. Let = VB{wi), Wi — (T:^^{wi — 1). The following conditions on the row 
exchangeable weights are assumed: 

Al. Eb{wi) = 1 

A2. 0<k <al^ o{n) 

A3, cin = Cov(wi,W2) = 0{n~^) 

A4. Conditions of Rcsultdl]) hold with Unj — Wnj- 
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Theorem 1 Under the conditions (Al)-(A^) on the weights, 

^a-\dl - e^) < x\Xi, ...,X„j~P[Y <x]= op(l) Vx- G R 
where Y ^ N{0,{1~ e^)). 
Proof Note that 

YJi=2 wtXt^ijext-i + Zt) 

J2wtXt-iZt 
' ^ T.^txu 

r, YjXtXt^i j2Xt-iZt 

_ Y,wtXt-iZt _ J2Xt-iZt 
" Ev^tXl, Y.XU 

_ Y^WjXt-iZt J2Xt~iZt J2Xt~iZt J2Xt^iZt 

EmXl, EmXl.^EmXl, Exl, 



Similarly 



Hence 



Now using Result 



"YliXt-lZt a.: 



n 

2 72 



Y.xUY.^tXU 



E{Xt-iZt) = 



Claim 1. Forr^ = a'^{l-9^)~\ 



Pp. 



1 " 



Xi , . . . , X„ 



A$( - ) Vxe 

T 



(7) 



(8) 



(9) 



To see this let us verify the conditions of ResultJT]) with c„j = XjZj^x and C/„j — Wj for j = 
1, . . . ,n - 1. 



1- ^I]"=2^t-l^t 

Follows from JHl 







2- ^Er=2^.'-i^?^^'(l-^')-H=r2) 
Follows from (jO)). 



3. n-i max(X2_^Z2) ^ 
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Proof Let Yt = Xf_j^Zf = XfXf_^ - 20Xt Xf_^ + e'^Xf_^ 
Then given e > 0, 

P{n~^ TciayiYt > e) = P(maxyt > ne) 



< Em>-) < 
t=i 



sy/ — > 



as = E{Xt,Zt) < 



Conditions (4), (5), (6) and (7) follow from definition and condition on the weights. This proves 
the claim. 



Hence for — a'^{l — 



P 



-^-Y^WtXt^^Zt < 
Claim 2. With c = - 9"^)-^, 

-jZ^tXU 

Tt ^ ^ 



, . . . , Xn 



> e 



A$( - ) Vxe 

T 



Ve > 



Proof 



EBi'-Y.-^tXl,) = '-Y^xi, 



s 7^ t 



Therefore 



s ^ t 



(10) 



(11) 



^Y^Xl, ^ E{X 



Hence the first term in dTTI) 
Also 



4 E ^ 



s ^ t 



Hence ^ _^ ^ is bounded a.s., and as ci„ — > 0, 

the second term in pT|) also — !• a.s. 
This shows that Vb Wt-'^t^-i) — ^ ^-S- 



Hence i E ^tXti - ^ E ^t-i ^ a.s. 
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Using Result ©, ^ E^t-i ^ E{Xf) = a^{l - 9^ 

This implies, ^ E«^t-^t-i ^ ^^^(1 ^ ^^)~^ a.s. 
This proves Claim 2. 

In fact we have proved that, with c = a'^ {1 — 6'^)~^ 



1 " 

79 ^ 



t=2 



> e 



Ve > 0. 



Now 



j:wtXt-iZt/v^ 

Ti - r2 (say) 



Then from ^ and ([12]), Pb{Ti < x) - P{T < x) = op(l), where, 

1 



cr2(l-6l2) 

Claim 3. Define A = y^{e„ - 0)a-^^J2iwt - l)^t-i 



(12) 



(13) 



(14) 



Then Ve > 0, Psi \A\>e) 
Proof Note that, 



0. 



Eb{A) 
VsiA) 







n 



■5 # t 

s2 t 



(say) 



— — ^ converges a.s., and from {6n — 0) — > 0, as a result, Ai — > 0. 



Moreover 



(15) 
(16) 

(17) 
(18) 



is bounded a.s., nci„ is bounded and cr^ ig bounded away from 0. As a 



result ^2 — ^ 0. 
Combining, VsiA) 0. 
Hence 



PBi\A\>e) < 



Vb{A) p 



Now To = 



A 



From (1121) . we have, '^WtX^_i/n is bounded away from zero in Pb a.s., which means that, Ve > 0, 
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^^5(1^21 >e) = op(l) (19) 
Hence from ([131), ([HI) and ([H]), we have, 

Pb[ V^a^^iO*,, - e'n) < X ] - P[ Y < X ] = op(l) Vx e M (20) 

where F - iV( 0, (1 - 9^) ) and this was what was to be proved. 

1.2 Least Absolute Deviations Estimator 

Another estimator of Oq can be the LAD estimatior, that is. 



1 " 

02 = arg min — > 



e n 

2 



Now we reparametrize the model ([TJ in such a way that the median of Zt, instead of the mean is 
equal to 0, while VZt = remains unchanged. 

1.3 Distributional Consistency of the LAD estimator 

Under the following assumptions we establish the assymptotic normality of 62- 
Al. CDF Zt, F has a pdf /, which is continuous at zero. 

A2. I F{x) — F{0) — a;/(0) | < c|x|^+" in a neighborhood of zero, say \x\ < M, where c , a ,M > 0. 
To do so we use the the following result on random convex functions. 

Result 3 (See Niemire (1992)) Suppose that hn{a), a E R'^ is a sequence of random convex func- 
tions which converge in probability to h{a) for every fixed a. Then this convergence is uniform on any 
compact set containing a. 



Theorem 2 Under the conditions (Al)-(A2), \fn{()2 — 0o) — > N{0, jj^^Jex^) as n 00. 
Proof Define 



f{Xt,9) = {\Xt-0Xt-i\ - \Xt\) 

g{Xt,e) =. Ai_i[2/(Zt(^?) <0) - 1] 

where Zt{0) = Xt - 0Xt^i fort = 2, . . . ,n. 

Yt{a) = f{Xt,0o + n-^^^a) - f{Xt,0o) - n-^/^ag{Xt,0o) 

= \Zt~n-^l^aXt-x\ - \Zt\ - n-^l^aXt-x\21{Zt<^)-\\ for o, G 



Also define 



,(0) = 

C/„ = ^g(X*,0o) 

K = ^yt(a) -Q„(0o + n-i/2a) - Q„(0o) - n-i/2aC/„ 



Stepl YTt=2^M) A a^f^m^l 
Stepl.l ^(y* - E{Yt\At-i)) A 
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E{ Yt - E{Yt\At-i) ) = 
ViJ^Yt- E{Yt\At-i)) = ^V{Y,- E{Y,\At-^)) < ^V{Y,) < Y,EY,^ 

By convexity of /, 

< Yt{a) < n-^/^a[g{Xt, Oo + n'^/^a) - g{Xt, 60)] 

Therefore 

2 
n 

= 4-EXl,[I{Zt - n-'/^aXt-i < 0) - I{Zt < 0)]^ 



E{Y,^) < ^E[9iXt,eo + n-'/^a)-giXt,eo)]'' 
n 

Now 



EY^ = nEYi < AaEXl[I{Z2 - n'^'^aXi < 0) - < 0)]^ 

which tends to zero using DCT. Therefore 

v{Y,{yt - E{YMt-i))^Q 

This establishes Step 1.1. 

Stepl.2 E E{YiAt-i) - a^f{0)EXl A 

E{Yt\At-i) = E{\Zt-n-^'^aXt-i\\At^i)-E\Zt\ 
= j{\z-n-^'^aX,_^\-\z\)dF{z) 

Using the representation, 

\x-e\- \x\ = 0[21{x < 0) - 1] + 2 [ [I{x < s) - I{x < 0)]ds 

Jo 

we have 

\z - n-^/'^aXt-i\ - \z\ = n-^/'^aXt-i[2I{z < 0) - 1] + 2 / [I{z < s) - I{z < 0)]ds 

Jo 

Therefore 

[21 {z <0)-l]dFz + 2 J J [I{z <s)- I{z < 0)]dsd^i) 

= 2 [F{s) - F{0)]ds (22) 

Jo 

= 2n-^/^Xt-i r[F{n-^/^Xt-ix)-F{0)]dx (23) 
Jo 

Under assumption A2, 

F(n-i/2Xt_ix) - F(0) - n-i/2Xt_ia;/(0) + i?„t(x) 
where \Rnt{x)\ < cn-(i+")/"|Xt_i|i+"|a;|i+" 
whenever n~^^'^\Xt-i\\x\ < M 
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Hence 

E{Yt\At-i) = 2n-^'^Xt^i I [n-^/^Xt^^xf{0)+Rnt{x)]dx 

Jo 

= -Xf_^a^fiO) + 2n-^/^Xt-i [ Rnt{x)dx 
n Jo 

n n Jq 

= h+ h {say) 

Then h A a^f{<d)EXl. 

Remains to show I2 — > 0. To show this, let us assume: 
1. maxi<t<„ n"i/2|Xf_i| 

Hence given e > 0, 

P{-ai&yi n-^l'^\Xt-i\<M/\a\) 1 

Let An be the set where max n-^/'^\Xt-i\ < M/\a\ and ;^rfe^ E |^t-iP+" < e. 

Then 37V such that P{A„) > 1- eV n> N. Then on A„, |i?„t| < cn~"/2|Xt_i|i+", and hence 



I/2I < - Vl^t-il rcn-"/2|X,„i 
" Jo 



l+a 



y2-L-r(:K/2 

< e 

ieP(|/2|<e)->lVe>0. In other words I2 — > 0. This completes Step 1.2 and hence Stepl. In 
other words, 

QniOo + n^'/^a) - Qn{Oo) - n-^/^aU-a - a^f{Q)EXl A (24) 

Due to convexity of Qm the convergence in (p4)) is uniform on any compact set by ResulllSl Thus V 
e > 0, and M > 0, for n sufficiently large, we have 



P 

Call 



sup 

|a|<A/ 



ln{eo + n-^'^a) - QM - n-^^\Un-a^fiO)EXf 



An{a) = Qn{0o+n~^/^a) - QM 
Bn{a) = n-^/^aUn + a^fiO)EX^ 

and their minimizers a„ and 6„ respectively. Then 

a„ = \/n{92 — do) and 

bn - -{2f{0)EXf)-^n-'/^Un 

The minimum value of i3„, 

Bn{bn) = -n-\Af(Q)EXl)-^Ul 



< e 



> 1 - e/2 
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Note that 6„ is bounded in probability. Hence there exists M > such that 



P 

Let A be the set where, 



I - {2f{Q)EXl)-^n-^/^Un\ <M -I > 1 - e/2 



sup \An{a) - Bn{a)\ < e 

\a\<M 

and I - {2f{0)EXf)-'^n-^^^Un\ <M-1 

Then P{A) > 1 - e. On A, 

A„(6„) < B„(&„)+e (25) 

Consider the value of A„ on the sphere 5„ = {a : |a — = fce^/^} where k will be chosen later. By 
chosing e sufRciently small, we have \a\ < M Va ^ Sn- Hence 

An{a) > Bn{a) ^ eVa e Sn. (26) 

Once we chose k = 2{2f{0)EX^)-^^^ , 

B„(a)>H„(6„) + 2eVa £ S„ (27) 

Comparing the bounds (1^ and we have A„(a) > ^„(6„) whenever a € S^. If |a„ — 6„| > /ce^/^, 
by convexity of A„, thereexists a* on such that yl„(a*) < A„(6„) which cannot be the case. 
Therefore |a„ — 6„| < ke^^'^ on ^. Since this holds with probability atleast 1 — e and e is arbitrary, 

this proves that |a„ — 6„| — 0. In otherwords, 

V^i02 - do) = -n-'/\2fiO)EXf)-'Un + op(l) (28) 
Step 2 n-i/2C/„ ^ 7V(0,£;X2) 



Un = ^Xt_i[2/(Zi <0)-l] 
n 



t=2 

Then note that C/„ is a 0-mean martingale with finite variance increments. Hence to prove Step2, we 
use the Martingale CLT. Write 

n n 
t=2 t=2 

and si = ESl = {n- l)EXl 

Then we need to to verify: 

1. % ^ 1 

This follows from Result [2l 

2- Y,"^2 E{Y^I{\Yt\ > esn)) ^0 as n ^ Ve > 0. 
To see this, note that 



as EXl < oo 
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Hence using Result H we have ^ iV(0, 1), which proves Step2. 
Combining Step2 and eauation dj^ . we get, 



and this was what was to be proved. 
Finahy it remains to verify: 

1. max2<t<„ 
Proof: Given e positive, 



P(maxn-i/2|Xf_i > e) < '^P{\Xt\ > eV^) 

t=i 

= {n~l)P{\Xi\> eV^) 

= (n-l) J Ii\Xi \ > eV^)dP 

< {n-l) ( > e^/^dP 



^ J \Xfl{\Xi\>eV^T)dP 
OasElXif <oo 



Proof: 



^ V^ITT |2+a / maxi<t<„_i |Xt|" 1 2 

;^Tw^Z^l^*-il - ;;t+^ ~Z^^*-i 



< 
p 



max iXt 



This follows from (1) and the fact that -^J^-^t-i is bounded in probability, since EXf < oo. This 
completes the proof. 

1.4 WBS for LAD estimators 

Now we define the weighted bootstrap estimators, 62 of 62 as the minimizers of 

n 

QnB [e] =Y,Wnt\Xt- eXt-l\ (29) 
t=2 

In the next section, we deduce the consistency of this bootstrap procedure. 

1.5 Consistency of the Weighted Bootstrap technique 

Now we prove that the Weighted Bootstrap estimator of O2 is assymptotically normal with the same 
assymptotic distribution. In particular WB provides a consistent resampling scheme to estimate the 
LAD estimator. 
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Theorem 3 Let 62 be the weighted bootstrap estimator of 62 as defined in 



Suppose the bootstrap 



weights satisfy conditions (A1)-(A4). Also assume that n ^/^cr„ max( \Xt \ 0. Then 



sup 



P 



-62)<X 



Xi, . . . , Xr, 



- P[Y <x] 



= Op{l) 



(30) 



where y - iV ( 0, 4j.2(o^)i;xf 



Proof Define 



UM = f{Xt,0o + n-'/^ana) - fiXt.Oo) -n^^^^anagiXt.eo) 

UnBt[a) = WntUnt{a) 

SnB = ^M/„t5(Xt,0o) 

Snw = y^^WntgiXt,OQ) 

H = 2f{Q)EXl 



Then 



EsUnBt = Unt and 

UnBtia) = QnB{0a + 7i"^/^cr„a) - Q„b(6'o) - n^'^/^anaS„n] 

Stepl. We show V"cr,7^(^2 - ^2) = -W^/'^H-^SnB + TnB s.t. given e > 0, PB[\rnB\ > e] = op{l). 
To show this, choose k = 3H~^^^ and e small enough such that k^e < 1 and M a sufhciently large 

constant. Let A be the set where 



sup cr„ 

\a\<M 



inB 



< € 



and 



<M-1 



Then due to convexity of QnB, arguing as in the proof of Theorem2,on A we have, 

V^cT-\e; - Bo) = -n-^'^a-^H-^SnW + r„s 
s.t. |r„s| < ke^/^ 

If we show 1 — -Pb[.4] = op(l), then 

PB[\rnB\> 5]=op{l) V<5>0 

Also from equation ipg)) : \/n(^2 - ^'o) = -n^'^/'^H^^Sn + op(l). 

Therefore y^a-^{6'^ - §2) = -n-^/'^R-^SnB + ?'nB2 s.t. given e > 0, PB[\rnB2\ > e] = op(l). 
This will complete Stepl. 

Hence it remains to show, 1 — Pb[^] = op(l) 
To show this we show, 



V M > 0, P; 



n 



Y,UnBt{a)~^a^H 
and there exists M > s.t. Pg [|cr,7^?i" 



M 



= Op(l) 
= Op(l) 



(31) 
(32) 
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To show (PT|) . note that, 



Pr 



11 



> e 



< E PbK'I E WtUt{b,)\ > e/2] + E /(^n'l E ^*(^^) ~ ^n^'^/2| > 

< E ^ E ^'(^^■) + E E ^*(^^-) - > e/2) 

As a result, we need to show for fixed b, 

^n'Y.Untib) = Op(l) 
t 

and(T-2[^[/„,(5)-a2 62ii-/2] = op(l) 



(33) 
(34) 



To see l|33|). 



This proves ([33 



= na-2i?C/f(6) 

< Eb^[g{Xi,0o + n-^^^anb) - g{Xt, 0o)]' 




To prove ([M)) note that, 

^n' [Et^t(^)-^n&'^/2] [E[C^*(^)-m(6)|A-l)]+E^(t^*(^)l-^*-l)-^n^'^/2 

E(f^*(^) - EiUtib)\At-i)) = 

^K'E(f^*(^) -^(f^*(^)l-^*-l))] = <^n'Y.^iUt-EiUMt-l)) 

< a-*j2^mb)) 

< fcr'<'E^(^'(^)) (^n>^i) 

= nk^'a-^E{U,{b)f 

< k^'a-^alb^E[g{Xi,eo + ji-'/^ar,b)-giXi,eo)]'' 

= ^E[g{Xi,0o + n-^/^a,,b) - g{Xi,9o)]^ 
ki 



Hence 



<y-'[Y.(Ui(.b)^E{Utib)\At-i))]^0 



(35) 
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(7-2 V E{Ut\At^i) = (T-2 V 2n-^'^Xt-i / [F{n-^'^anXt-ix) - F{Q)]dx (36) 



JQ 

2n-i/V,-i^Xt_i n-i/V„yXi_i/(0) + i?„t 
where |i?„t| < c|n-i/2^^x^_ = cin-'/^a,,y+''\Xt-i\'+" 

= h+ h (say) 



(37) 

(38) 
(39) 

(40) 
(41) 



Here follows from and ([55)1 from assumption A2 on F and the assumption n ^I'^On maxj \Xt\ 
0. 



iVow /i ^ \?j{<S)EXl = h'^H/2 

and \h\ < c(n-VV-i)(n-VV„)i+"5]|X,_iP+" 

_ ''"ri \ ^ I -v^ \2+a 

A 

In this case (05]) follows from dH]) if we show ^ji^ E Va > 0. 

To see this, note that 



n 



<maxt|Xt|" 1 



< 



^ max \X,\Y-y2xf 



(42) 
(43) 

(44) 

(45) 



Combining ^ and (gS]), we have cr-^ ^ £;([/j|ylj_i) -A 62iy/2. In o.w., 



[5^i?(t/,|A-i)-cT^62i//2 







Adding ^ and gS]), we prove And from ([Ml) and ^ we deduce ([HI 



(46) 



Now P, 



B 



< 

< 
p 



Eb \^wtg{Xueo) 



M2n 







if M is choosen sufHciently large. This proves (|32|) . (|3T1) and (|32|) together show 1 — Pb [„4] = op(l). 
This completes step 1. 
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Step2. PB{n-^'^SnB < x) - P(r < x) ^ op(l), where Y ~ N{0,EXl) 
To show this we use Result 1. 



SnB - Y.^^i9{XuOo) 

= J2WntXt^i[2IiZt<0)-l] 



Hence we need to show: 



1. iE^t-i[2/(^t <o)-i] Ao 

3. imaxtX2_i A 
All these follow from Step2 in the proof of Theorem2. 

This completes Step2, and combining with Stepl, we get 

Pb [V^<Jn\^*2 - S2) < a;) - P{Y < x) = op(l) V x e 

where Y ^ N \ ^ 



Using continuity of the normal distribution, we complete the proof. 
1.6 Special choices for w. 

With {wi, . . . ,Wn we get the Paired Bootstrap estimator. This is same as 

resampling w.r. from (Xt_i, Xt), t = l,2,...,n. Other choices of {wij's yield the m-out-of-n 
Bootstrap and their variations. In particular lets check the conditions on the weights in two particular 
cases. 

Case 1. (wi, . . . , w„) - Mult(n, i, . ■ . , i) 

Clearly the weights are exchangeable. Let us verify assumptions (A1)-(A4) on the weights in this 
case. 

Al. Eb{wi) = 1 

Obvious in this case. 

A2. <k <(7l = o(n) 

(T^ = 1 — i which clearly satisfies the above condition. 

A3. ci„ = O(n-i) 

cin ~ which is as above. 

A4. {Wi} satisfy conditions of P-W theorem. 

To show this, we have to verify conditions (6) and (7) of Result ([!]) with Unj = Wj. 

Condition(6) ^^Y^Wf-^l 



Therefore, 



* n-ln ^ ' 
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- If) = nVBiwi - If + n{n - 1)Covb{{wi - if, {w^ - if) 

Write wi = Ui and W2 = X^^Li ^i, where {uj, Vi}f^i are iid with the joint distribution of 

{ui,Vi) given by 

(1,0) w.p. 1/n 
{ui,Vi) = { (0,1) w.p. 1/n 

(1,0) w.p. 1-2/n 

VB{w,-lf = Eb{w,-1)*-V^{w,) 



EB{w^-l)'-^{l--f 



{wi - 1) = Z^"=i(wi - p) where p=^,q=l-p. Hence 

Eb{w,-1)^ = E{Y.{ui-p)^ + 2Y.^Ui-pf{uj-pf) 

i 3 

= nE{ui-p)^ + 3n{n-l)p'^q'^ 



Simphfying 



Therefore 



n{pq +p q) + 3n(n — l)p q 



1 Q fi 2 

= (l-i)(4-%A + ^) (47) 



^b(w;i-1)' = (1--)(3-- + ^ + ^)^3 (48) 



{w^-if{w2-if = [Y.iu.-p)r['£iv^-p)r 

X [J2(vi-pf + Y,J2(vi-p){vj-p)] 



Eb{Wi - lf{w2 - If 

E[Y,{u.-pf{vi-pf + ^Y.i^,-pf{vj-pf 

i iyij 

nE{ui - p)'^ {vi - p)"^ + n{n-l)V{ui)V{vi) 

+ n{n — l)Cov{ui, vi)Cov{u2, V2) 

n(2pp2g2 _^ p4(l_2p)) + n{n-l)p^q^ - n(n - 1)/ 

2 , 1,, 1 , 2, 
— {l--f + — 1-- 
n"" n n"* n 

3 4 3 
1 - - + — - — 
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CovB{{w^-l)Mw2-lf) 
EB{w,-lf{w2-lf - (1 



n 



3 4 
-- + — -- 

n 71^ ; 

1 3 3 

^ 9" ^ 



Therefore 

Fs(-VK-I)') = -yBK-l)2 + (l-i)Co«s((ii;i-l)2,(z. 
" -"^ — ' n fi 





n -"^ — ' n — 1 n ^ — ' 

n ^ — ^ n — 1 



Hence from (1491) and fl5Qll . 



2 -Ps, 



This proves condition (6). 



Condition(7) hm^^^oo hmsup„^^ E{Wm\Wt\>k)) = 



E{W^I\wA>k) = 4^[(^*-l)'^(l-*-i|>fe--.)] 

< ^[Eiwt ~ l)']-^ [Pi\wt ^ l\ > ka„)] 

1 10-2 

< -t(M„\)(- " - 



1, M„4 si 



where MnA = E{wt — 1)*. Therefore 



Urn limsup J E{Wt^I(^iWt\>k)) 

/c— >-oo ri— >-oo * 

< lim lim sup — p ( — J— J <i = 



as both Mn4 and cr„ are bounded (follows from ^47\ ). 
Case 2. (wi, W2, . . . , Wn) iid(l,cr^) 

Again we need to establish (A4), that is, verify conditions 6) and 7) in Result(IT]). 
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Condition 6) follows from WLLN. 

To verify condition 7), note that since distribution of (wi, W2, . . . , w„) is independent of n, 

lim VmYnn^ J E{W'il(\Wt\>k)) 

k Voo n-^oo * 

= lim Ji?(W^2j ) ^ 

k—^OD V 

since EW^ < oo. 



Remark 1. Result 2 is true even when the process is nonstationary. This follows from the fact that, 
given observations {Xt} from the AR process, Xt = 9Xt-i + Zt, \9\ < 1; we can get a stationary 
solution of the above process, say {Yt}, such that ^ S^f^t+fc ^O^t'^t+k)- 
As a consequence, Theorem 1) holds even without the assumption of stationarity, which is assumed 
throughout its proof. 

2 Bootstrap in Heteroscedastic AR(1) model 

Now we introduce heteroscedasticity in the model ([ij, and study the Weighted Bootstrap estimator. 
Consider the following model: 

Xt = 9oXt-i + Zu Zt^Ttet i=l,2,...,n. \9o\ < I (51) 
Xq ^ Fq with all moments finite. (52) 

where ^o, Tt > are constants, ^ iid{0, 1), and et is independent of {Xt^k, > 1} for all t. 



2.1 Estimation 

Based on observations Xi,X2, ■ ■ ■ , X^ we discuss various methods for estimating 9 in the model. 
Listed below are four types of estimators. 



(a) Weighted Least Squares Estimator Assuming {ti} to be known, consider the following 
estimator for 9q: 

1 " 1 

9i = argmin0-'^^—{Xt - 9Xt-if (53) 



t=2 * 

1 



t=2 

If et in model (|5ip is normal, (|54p turns out to be the (Gaussian) maximum likelihood estimators 



(54) 



(b) Least Squares Estimator In general {ti} are unknown and are non-estimable. Hence we may 
consider the general least squares estimators, ie. 



I]t^2 XtXt-i 

t=2 



(55) 
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This turns out to be the same as ([5^ if the {r^} are ah equal, that is the model is homoscedastic. 

(c) Weighted Least Absolute Deviations Estimator The estimators ([M)) and (|55|1 are L2- 
estimators. It is well known that Li-estimators are more robust with respect to heavy-tailed distri- 
butions than L2-estimators. This motivates the study of various LAD estimators for ^o- Now we 
reparametrize model (|5ip in such a way that the median of et, instead of the mean equals while 
Vet = 1 remains unchanged. Our first absolute deviation estimator takes the form 

" 1 

^3 = argming "S^ — \Xt - BXt-i\ (56) 

— Tf 

t=2 ^ 

This is motivated by the fact that 63 turns out to be the maximum likelihood estimator when the 
errors have double-exponential distribution. 

Least absolute deviations estimator Estimator ((56| uses the fact that Tf are known. Incase they 
are not our absolute deviation estimator takes the form 

n 

O4 — argming \Xt — 0Xt-i\ (57) 

t=2 

In the next section we discuss the assymptotic properties of the listed estimators. 

2.2 Consistency of estimation in heteroscedastic AR(1) process 

In this section, we establish the distributional consistency of each of the four estimators discussed in 
the earlier section. To do so, we will use some established results, the first one being the following 
Martingale Central Limit theorem: 

Result 4 (Martingale C.L.T.; see Hall and Heyde 1980) Let{Sn,J^n} denote a zero-mean mar- 
tingale whose increments have finite variance. Write Sn — X^ILi "'^^ ' — -^("'^i^-il-^i-i) 
and si = EV^ = ESl . If 

n 

s-^^E{Xfl{\Xi\>eSn)) — > as 00 \fe> 0. 

i=l 

Then ^ ^ N{0,1). 

Another result we will need is the following one on convergence of a weighted sum of iid random 
variables. 

Result 5 Let Ai, A2, . . . , A„ be a sequence of iid mean zero random variables, and {cj;„|« = 1, . . . , n} 
a triangular sequence of bounded constants. Then ^ CmXi — ^ 

2.2.1 Distributional consistency of 61 

Theorem 4 Define ^ — -^^1=2 n^t''^ ^-^t-i- Assume that 

Al. ^ < AhV 1< i < j < n 

A2. ii:^<,,<^.<„0f-)(^)2>Mi>O 
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n 2^1<i<]<n'^0 (tj) 



d 



Then under assumptions ( A 1-A2), Sn{di — da) — > N{0, 1) 
Further if we assume (A3), we have, ^/n{9i — 9q) N{0,9l/ p^) as n oo 



Proof y/^iOi - Oo) = 



Stepl. ^ rj^^X(_iZi is assymptotically normal 



Let 5™ — J2t=2'^t '^^t-iZt- 
Note that 



Xt = 0*Xo + ^0*-'=Zfe Vi > 1 (58) 
fc=i 

t 

Hence E{X^) = fl^t^^s ^ ^ ^2t-2fc^2 (gg^ 



fe=i 



Hence S'„ is a mean An measurable martingale with increments having finite variance, where 
At = fT(Xo, ei, £2, • ■ • , Et); ^ — l,2,...,ri. This follows from the fact that E{Xf) is finite, and 
E{Xt-iZt\At-i)=G. 

To establish the assymptotic normality of Sn, we use Result dH). Let 



t=2 

Then to accomplish Stepl, we need to show 



A 1 (60) 

si 



^ [{r^'Xt-iZtfl{T^^\Xt^^Zt\ > esn)] (61) 

To prove ((60)) . note that 



t=2,n 



^ n 

n n 

t=l,n-l 

t=l fc=0 *+l t=l,n-l 0<i<j<t *+l 

1 "^1 /n— 1 2 \ o /ri-1 \ 

fc=0 \t=fc / 0<i<j<n-l yt=j *+^ y 

= Ti + 2r2 (say) 
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Using assumption (Al) and Resul[Sl we have Ti 0. 



ET2 = 

n 

r,2t-i-j TjTj 
0<i<j<ri t=j *+l 



'''^^ = h ^ ^ 

0<i<j<n 
t=l \fc=0 



Hence T2 0. 
Combining, — — > 

n - n 2^t=l l^k=0 T^^O 



Using assumption(A2), is bounded below. Therefore ^ — > 1 and this proves (|60l 
Remains to show ([61]), ie ^ [Tt+^i-'^t^e^+i/(Tt+\|A:tet+i| > es„)] — ^ 



t 

fe=0 



*+i fc=o *+i o<j<i<t *+i 

Hence for 1 < < < n, 

S [T-^\X?e^,^,I{T,~-_^\\Xtet+,\ > es„)] 

k=0 *+l 0<i<j<t *+l 

- ^1 n'J^?^^.^ [4e?+i-^(Tt+\l^tet+i| > esn)] + ^2 „ max E [eieje'^^J{T^r^^^\Xtet+i\ > es„)] 

U<A;<r 

is bounded below by say, AI > 0. Hence for a fixed fco, < fco < i. 
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E 

< E 

< E 

< E 
= E 

< E 

< E 

< E 

< E 



2 2 r f\Xtet+l\ ^ 



k=0 



Tt+1 



,2 



+ E 



\k=0 



n+1 



E 



\k=0 



4l{\ei\>cin'/* 



e?/(|ei|>cX/4 



e?/(|ei|>cini/4 



e?/f|ei|>C3ni/4 



E 



E 



E 



4ol[T.\(^ot''\ek\>C2n 



1/4 



Vfc=0 



E 



k=iika 



£1^ 1 ^ 



C4 

ni/4 



Hence maxo<fe<t E [ele^^il {T^j^-^\Xtet+i\ > es„)] — > 0. Using the fact that ^ is bounded below, this 
proves del])- Using ResuUHl from §0^ and ([61]) we deduce that N{0, 1), ie 



1 " 

— Vrr'Xt-iZt AiV(0,l) 



(62) 



Step2. 



This follows from jBO 



Y^" ^-2 -^2 



(63) 



From and ([55)) we deduce, 



ie s„(^i - 0o) A iV(0, 1) 



(64) 
(65) 



n 



t=2 
^ n— 1 i 



t=l fc=l 



't+1 



(66) 
(67) 
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Hence if we assume (A3), we have ~^ fr, and then, 



V^{ei - f?o) ^ N(o, -|) 



This completes the proof. 



Remark Assumptions (Al) and (A2) are satisfied if {Tt}'s are bounded, or more generaUy if they 
are of the same order, ie there exists constants ci, C2 > and a > such that cit" < r*^ < C2<" for 
\<t<n. 

2.2.2 Distributional consistency of 62 

Theorem 5 Define = X]t=2 '''t^-^i'^t-i)- Suppose {tj} 's satisfy the following assumptions. 
Al. Ml < Tt < M2; t = l,2,...,n 
A2. ^ t2 > 

A3 iV T^T^fl^^-'"*^ -4 /j2 

n Z^l<i<j<n '^O ^ A' 

T/ien under assumptions(Al) and (A2), ^ V^(^2 - 6I0) ^ (^0, -^i^^) . 

Further if (A3) holds, ^{62 - 60) ^ N (o, ^'^^Vgf • 

Proof ^{§2 - 9o) = V^^jcff' 

Stepl. ^t-iZt is assymptotically normal. 

Let S„ = J2t=2 -^t-iZf- 

Then S'„ is a mean An measurable martingale with increments having finite variance, where 
At — cr(Ao, ei, £2, • ■ • , et)i ^ — l,2,...,n. This follows from the fact that E{Xf) is finite, and 
E{Xt-iZt\At-i)=G. 

To establish the assymptotic normality of Sn, we use Result (|4]). Let 



n 
t=2 

Then to accomplish Stepl, we need to show 

P 

A 1 (68) 

n 

— Y,E [{Xt_iZt)H{\Xt_iZt\ > esj] ^ (69) 

t=2 

Using the expressions for Xt and EXf from equations ([58l) and (|59)) . we have 
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^ n 






n n 






^ 71— 1 
t=l 






^ n— 1 t 

71 E E 

t=l k=0 






n— 1 
k=0 


-1) 


/n-l 
\t=k 



2 n-1 



71 

t=l 0<i<j<t 



0<i<j<n-l \ *=J 



2 n2t-i-j 
i 'J 't+l^O 



= Ti + 2T2 {say) 
Using assumption (Al) and ResullSl we have Ti — > 0. 



ET2 = (70) 

2 



VT.-^. E (e^o 

0<i<j<n-l \ t=j 



2t-j-j" 2 

'''i''j'''t+l 



(71) 



^ E El^ol"^^"^' (72) 

0<i<i<n-l t=j 

n-1 / t \ 2 

^ ;J2E EN-M ^0 (73) 



71^ 

4=1 \fc=0 



Here c is some positive constant. (|72p follows from ([7T|) using the fact that X]"=j ^0 ' '''i'''j'^t+i 
bounded which inturn follows from assumption(Al). 

Hence T2 0. 
Combining, ^ — ^ 

n n 

Also — i V""^ V* Tfc /i2(f-fc) 

^^^O n - n Z^t=l Z^fe=0 T^'^O 

Again using assumption(Al), is bounded below. Therefore — > 1 and this proves 
Remains to show ie p- E"=i^ ^ [Tt+i-^t^e?+i-^(Tt+i|A:tet+i| > es„)] — > 



t 



fc=0 0<i<j<t 
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Hence for 1 <t<n, 

E [T^+^Xlel+J{Tt+i\Xtet+i\ > es„)] 
t 

k=0 0<i<j<t 

< Ai max E [elet^iI{Tt+i\Xtet+i\ > es„)] + A2 max E [eiejet^iI{Tt+i\Xtet+i\ > es„)] 

is bounded below by say, M > 0. Hence for a fixed fco, < < ^, 

i?Ke?+i/(rt+i|Xtet+i| >es„)] 

< E [ele^t+il {Tt+i\Xtet+i\ > eAlV^] 



< E 



< E 



E 



< E 



< E 



< E 



< E 



E 



{\et+i\ > VTFln^'^) 

e?/(ki|>Cini/4 
e?/(|ei|>cini/4 



E 



\fc=0 

4,/ (iZ''un+M''^\eu\ > V^n'A 



k=0 



E 



ell{T.\^or'\ek\>C2n 



1/4 



^fc=0 
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E 



E 



1/4 



£1/ (|ei| > c^n 



1/4 



C4 
r7l/4 



Hence maxo<fc<t E [ele^j^il {Tt+i\Xtet+i\ > es„)] — s- 0. Using the fact that ^ is bounded below, this 
proves 

Using ResuhH from dMl) and dH]) we deduce that ^ N{0, 1), ie 



Step2. 



This follows once we show 



1 " 

t=2 
1 " 



t=2 



iE(^*'-^^^')^0 

1 ^""'^ 

and - E 



n-l 



(l-^o') 



(74) 
(75) 

(76) 
(77) 
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fi2t—i—i 



Using the expressions for Xt and EX^ from equations ([551) and (ICT)) . we have 

^'j2iXf-EXf) 

t=i 

k=0 \t=k / 0<i<j<n-l \*=J 

The above steps can be justified by proceeding as in the proof of (|68)) . This complete ([76|. 
To see (ffT]). note that 

n— 1 77 2 -1 n—1 t 



^E^^* = ^E^o* + ^EE-^^^o*-" (78) 

t=i t=i t=i fc=i 

^EE-^X"-^' (79) 

^ n— 1 71 — k— 1 

^E-^' E ^0* (80) 



fc=l t=o 



^^"'^ (1-^0^) ^''^ 
(T^^E^.^ (82) 

(83) 



(1-^0^ 

This proves ([771) . (|76p and ([77[) together prove ([75]) and this completes Step2 
Dividing ([74l) by ([75]) we deduce, 



- 00) A TV (o, ^^^) 



t=2 
^ n— 1 t 

„ E E 

t=i fc=i 

2 2 

Hence if we assume (A3), we have — — fr, and then. 



n-1 t 

2 2 /i2(t-/c) 

t=l fc=l 



^/^{02 - 0o) A N ( 0, 

This completes the proof. 
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2.3 Consistency of the Weighted Bootstrap technique 

Now we prove that the Weighted Bootstrap estiraator is assymptotically normal with the same assymp- 
totic distribution as of the least squares estimate. In particular WB provides a consistent resampling 
scheme in the AR model with introduced heteroscedasticity. 

Theorem 6 Let 0* be the weighted bootstrap estimator of 0„ as defined in Q). Then under the 
conditions (A1)-(A4-) on the weights, 



P 



Xi , . . . , Xn 



where Y ^ N{0,a'^), being defined in Theorem (11) 
Proof As in (Hg, 



P[Y <x\= op(l) Va; e 



(84) 



= TI-T2 

Claim 1. There exists t > such that 



1 " 

-^ywtXt^iZt < 



Xl, ■ ■ . , Xn 



-i E{m-l)Xl,/r 
Y.^tXf^Jn 



A$( - ) Vxe 

T 



(85) 



To see this let us verify the first three conditions of Result(IT]) with c„j = XjZj^i and Unj — Wj for 

1 P 



Condition 1 - I]t=2 ^t-i^t ~ 
Follows from Theorem ??. 

Condition 2 ^ EL2(^t-i^t)' 







Then Sn = Un + K- 



Let Sn — 



Un = 



Vn 



t=2 

'-±xl,iz!^af) 

n 



t=2 



E{Un) = 

71 

V{nUn) = Y.^{Xl,{Zf~a',)) + 2Y,Y.^^<^t-i(^t-'^tlXl,(Z^,-a'J) 



2<s < t<n 



= Y.^{xU{zl-ol)) 

n 

< M ^ EX^_-^ where EZf < M 
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Therefore V{Un) < —Y,EX^_^^0 since — ^ EXf_-^^ is bounded. 



Hence f/„ — > 0. 



t odd t even 

2 St odd -^t-l I 2 St eiJen -^t-l 
0^ + 1T2 



2 ^ 



n/2 



Therefore 



2(1 - 6»4) ' 

_P 1 

2 (1 - 6*4) 

^'(^i+f^2) + 2afa2 2 



2(1 - 6»4) 



Condition 3 n'^ max(X2_^z2) 
Given e positive, 



P(n-i max(x2_iZ2^ > e) < ^ P(yt > ne) where Yt = X'^_-^Z\ 

2 

1 " 



M 



f=2 



t=2 



since — -EX* is bounded. 



Tliis proves Claim 1. 



Hence jar r = - 



2(1 - 6»4) 



^VWtXt_iZt< 
vn f-; 



A*( - ) Vxe 

r 



Claim 2. 



Wit/i c = 



2 (1-02) 



> e 



Ve>0 



t=2 
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Note that 



n/2 n/2 

Using the fact that {Xt]teven and {Xt}todd form two homoscedastic AR(1) processes, from Claim 
2 (Theorem [T]) and Remark 1, we get, 



Y.toddWtXf-1 Ps^ e'^ol+al 



and 



Hence 



n/2 1 - 614 



a.s. 



n/2 



1-614 



This proves Claim 2. 



Claim 3. 



Pb 

This follows from equations (??) and (fSTl 
Note that as defined in (|85|) . 

Then from iP]) and ([57|). 



1 " 



> e 



Ve > 



na-'ie: - On) 



To 



where T 



Pb{Ti <x) - P{T <x) = op(l) 



N 



2 (1-^2) 

Moreover using equations (1571) and (155)) . from Claim 3(Theorem[T]), we get, 

Pb{ \T2>e\ ) = op(l) Ve>0 

Combining 

Pb[ V^^<JnHK -On)<x]^P[Y <X]^ Op(l) Vx G 



where 



y - iV 0, 4 



(l-0^)afa22 + ^?2K + a4)/2 



(1 + 9^) K+ai) 



2\2 



and this completes the proof. 



(89) 



Remark 2. In Theorems 1 and 3, we have established the consistency of the Weighted Bootstrap 
estimator in probability, ie we have proved, V .t G K, 

Pb{ V^l<Jn\K - 0n) < X) - PiV^l{9n-9) < X) = Op(l) 

The same results can be achieved almost surely. One can prove that, V x G M, 

Pb{ V^<y-\9:, - On) < x) - P{^{9n-9) < x)^Oa.s. 

To prove this, one needs to verify the conditions of Result (1) almost surely, and replace all convergence 
of sample moments of {Xt} in probability, by almost sure convergence in the proofs. 
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3 Numerical Calculations 



In this section, we compare numerically the performance of the Weighted Bootstrap and Residual 
Bootstrap techniques for an hctcroscedastic AR(1) model, and exhibit numerically, the consistency of 
the Weighted Bootstrap estimator. We simulated 50 observations from the AR process, 

Xt = eXt-^ + Zt, t=l,2,...,n. 

where Zt is a sequence of independent Normal mean-zero random variables with EZ"^ = crj if f is odd 
and EZ^ = a^'^it is even. For simulation purpose, we used 9 = 0.5, al = 1, and a"^ = 2. 
The unknown 9 is estimated by its LSE 9n which came to be 0.4418. 

Let Vn = \fn(9n — 9) be the quantity of interest which is to be estimated using resampling techniques. 
Let V* = \/n{9'^ — 9n) denote its bootstrap estimate for two different bootstrap techniques: the 
Residual Bootstrap (which tacitly assumes that all the Zts have same variance) and the Weighted 
Bootstrap. In case of WB, we used i.i.d Normal(l,l) weights. We used 200 simulations to estimate 
the distribution of V* in both the cases. 

We performed the KS test to compare the distributions of Vn and V* . To estimate the distribution 
of V^, we used 200 simulations from the above process. The results of the test are as follows. 

Two-Sample Kolmogorov-Smirnov Test 

Data: Vn and V* 
Alternative hypothesis: 
cdf of Vn does not equal the cdf of V* for at least one sample point 



BS Technique 


KS value 


p- value 


RB 
WB 


0.12 
0.1 


0.0945 
0.234 



Figure la) presents the estimated densities of Vn and V^, with 9*-^ being the residual bootstrap estima- 
tor, while Figure lb) presents the estimated densities with ^* being the weighted bootstrap estimator. 
Prom the table it can be seen that both the estimators pass the test, but WB does reasonably better. 
This is also obvious from the density plots. 

Next we introduced more heteroscedasticity in the model. This time we took a\ to be 1, and (t| 
as 10. 9n came to be 0.47083. Again we estimate I^ by and performed a KS test to determine 
the goodness of the fit. Now the results are as follows: 

Two-Sample Kolmogorov-Smirnov Test 
Data: K and V* 

Alternative hypothesis: 
cdf of Vn does not equal the cdf of V* for at least one sample point 



BS Technique 


KS value 


p- value 


RB 
WB 


0.135 
0.125 


0.0431 
0.0734 



Figure 2a) presents the estimated densities of Vn and V* for RB, while Figure 2b) presents the 
estimated densities for WB. From the table, it can be seen that RB fails. This is expected since it 
is not adapted for heteroscedasticity. It fails to capture the true model in such a situation. WB still 
performs well, but its performance also falls. This is also reflected frm the density plots. Perhaps a 
larger sample size is required in case of substantial heteroscedasticity. 

This illustrates the point that for small sample sizes, at small levels of heteroscedasticity, many 
Bootstrap techniques perform well , but at substantial levels a careful choice is needed. The success 
of WB for both levels of heteroscedasticity lends further support to our theoretical results. 



29 



4 ARCH models 



In this section, we first present the basic probabiUstic properties of ARCH models. Then we in- 
troduce various estimation procedures for the parameters involved, and study their properties. The 
assymptotic properties of the listed estimators under different error distributions are also introduced. 
To approximate the distribution of the estimators and draw inference based on an observed sample, 
various resampling techniques are also listed along with their properties. Finally we supplement our 
theoretical results with numerical calculations based on a simulated ARCH data set. 

4.1 Basic Properties of ARCH Processes 

Defination 1 An autoregressive conditional heteroscedastic (ARCH) model with oreder p (> 1) is 
defined as 

Xt = atet and = cq + biXl_^ + . . . + bpX^_p (90) 
where cq > 0, bj > are constants, et ^ iid{0, 1), and et is independent of {Xt^k, ^ > 1} for all t. 

The necessary and sufficient condition for (l90t to define a unique stationary process {Xt} with 
EXf < cx) is 

<1 (91) 
Furthermore, for such a stationary solution, EXt = and V{Xt) ~ cq/{1 — J2f=i ^i)- 

4.2 Estimation 

We always assume that {Xt} is a strictly stationary solution of the ARCH model ([TO]) . Based on 
observations Xi, X2, ■ . ■ , Xn, we discuss various methods for estimating parameters in the model. 
Listed below are four types of estimators for parameters cq and bi . They are the Conditional Maxi- 
mum Likelihood Estimator, and three Least Absolute Deviations Estimators. 

(a) Conditional MELximum Likelihood Estimator If et is normal in model (j90|) . the negative 
logarithm of the (conditional) likelihood function based on observations ATi, . . . , X„, ignoring 
constants, is 



t=p+i 



(loga^Xf/af) (92) 



The (Gaussian) maximum likelihood estimators are defined as the minimizers of the function above. 
Note that this likelihood function is based on the conditional probability density function of Xp^i, . . . , X„, 
given Xi, . . . ,Xp, since the unconditional probability density function, which involves the joint den- 
sity of Xi, . . . , Xp is unattainable. 

(b) Least Absolute Deviations Estimators The estimator discussed in (a) is derived from maxi- 
mizing an approximate Gaussian likelihood. In this sense, it is an L2-estimator It is well known that 
Li-estimators are more robust with respect to heavy-tailed distributions than L 2 -estimators. This 
motivates the study of various least absolute deviations estimators for cq and bi in model (j90p . 
Now we reparametrize the model (j90[) in such a way that the median of , instead of the variance of 
et, is equal to 1 while Eet = remains unchanged. Under this new reparametrization, the parameters 
Co and bi differ from those in the old setting by a common positive constant factor. Write 

-l + eti (93) 



where eti = (e^ — 1) has median 0. This leads to the first absolute deviations estimator 



30 



= argmin ^ \X^I(Tt{9f - 1| (94) 
t=p+i 



which is an Li estimator based on the regression relationship (|93p . 
Alternatively, we can define another form of least absolute estimator as 

n 

§2 = arg mm ^ | \og{X^ - log(at {ef)\ (95) 
t=P+i 

which is motivated by the regression relationship 

\og(X^^)^\og{(Jt{ef)+et2 (96) 

where et2 — log(et ). Hence median of et2 is equal to \og{raedian{ef)\ , which is under the reparam- 
eterisation. 

The third L-1 estimator is motivated by the regression equation 

= + e« (97) 

where eo = cr^^e^ — 1). Again under the new parameterisation, the median of is 0. This leads to 
the estimator 

n 

03 = arg mm ^ \X! - at{e)^\ (98) 
t=p+i 

Intuitively we prefer the estimator 62 to 63 since the error terms et2 in regression model (jHH) are inde- 
pendent and identically distributed while the errors 643 in model (|97|) are not independent. Another 
intuitive justification for using 02 is that, the distribution of X^ is confined to the nonnegative half 
axis and is typically skewed. Hence the log-transformation will make the distribution less skewed. 
The minimization in (|94)) , (l95t and ((98|) is taken over all cq > and all nonnegative fo^'s. 

4.3 Assymptotic Properties 

In this section we discuss the assymptotic properties of the estimators listed above. 
The conditional maximum likelihood estimation remains as one of the most frequently-used methods 
in fitting ARCH models. To establish the assymptotic normality of the likelihood estimator some 
regularity conditions are required. Let {Xt} be the unique strictly stationary solution from ARCH(p) 
model (PU)) in which ct may not be normal. We assume that p > 1, cq > and bi > ioi i — 1,2, ... ,p. 
Let (cq, a'^)^ be the estimator derived from minimizing ((92)) . which should be viewed as a (conditional) 
quasimaximum likelihood estimator. 

Let 9 = (co,a'^)-^, 9 = (co,a'^)^, and Ut = It may be shown that Ut/u^ has all its moments 
finite. We assume that the matrix 

is positive definite. Further we assume that the errors are not very heavy tailed, ie E{e^) < 00. Then 
under the above regularity conditions, it can be established that (see Hall and Yao 2003) 

-9) N{0,M'^) 



{E{4) - 1)1/2 



If E{ef) = 00 the convergence rate of y/ii is no longer observable. Then the convergence rate of the 
likelihood estimator is dictated by the distribution tails of e^; the heavier the tails, the slower the 



31 



convergence. Moreover, the assymptotic normahty of the estimator is only possible if i!^(|et|^ ^) < oo 
for any 5 > 0. 

The asymptotic normality of the least absolute deviations estimator 62 in (|95p can be established 
under milder conditions. To do so we will use the reparameterized model. Let 9 = (co,a'^)^ be the 
true value under which the median of equals 1, or equivalently the median of \og{e^) equals 0. 
Define Ut and M as before. Again we assume there exists a unique strictly stationary solution {Xt} 
of model ([50]) with Eg{Xf) < 00. The parameters cq and &i, i = 1, 2, . . . ,p are positive. M is positive 
definite. log(ej ) has median zero, and its density function / is continous at at zero. 
Under the above conditions, there exists a sequence of local minimizers ^2 of (|95p for which 

V^{92 ^9)^ 7V(0,M"V{4/(0)2}) 

(see Peng and Yao 2003). Thus the least absolute deviations estimator 6*2 is asymptotically normal with 
convergence rate y/n under very mild conditions. In particular, the tail-weight of the distribution of 
£( is irrelevant as no condition is imposed on the moments of tt beyond E{e'^) < 00 

Similar to the above result, y^(^i — 9) is also asymptotically normal with mean 

E[etl{e-t > 1) - e?i"(e? < 1)] [E\mii\, . . . , i;|m(p+i)(p+i) |]^ where M = {m^jjij 

(see Peng and Yao 2003) which is unlikely to be 0. This shows that 9i is often a biased estimator. 

It can also be shown that \/ri{9^ — 9) is also assymptotically normal under the additional condi- 
tion EXf < 00. 



4.4 Bootstrap in ARCH models 

As indicated in the earlier section, the range of possible limit distributions for a (conditional) Gaussian 
maximum likelihood estimator is extraordinaily vast. In particular the limit laws depend intimately 
on the error distribution. This makes it impossible in heavy tailed cases to perform statistical tests 
or estimation based on asymptotic distributions in any conventional sense. Bootstrap methods seem 
the best option for tackling these problems. 

Residual Bootstrap(m-out-of-n) for likelihood estimator: Let 

it = Xt/<Jt{9) for i = p + 1, . . . , n and let {tt] be the standardized version of {?(} such that the 
sample mean is zero and the sample variance is 1. We define 



1 



t=i 



n ^ — ^ 



t=l 



Now we draw {e^} with replacement from {ct} and define X^ 
t = p + I, . . . ,m with 



(an' = co + E6.(x;_,)2 



<j;e* for 



and form the statistic {9* , t*) based on {X*^^, . . . , X^} in the same way as {9, t) based on {Xp^i, . . . , X„}. 
It has been proved that (Hall and Yao (2003)) as n — >■ 00, m 00, and m/n — > 0, it holds for any 
convex set C that 



G C\Xi, . . . , X„ 
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Weighted Bootstrap for likelihood estimator For every n > 1, let {wnt}, t = 1, . . . ,n, be real 
valued row- wise exchangeable random variables independent of {^t}. Then we define the weighted 
bootstrap estimators, 9* of 6 as the minimizers of 

n 

J2 w^t[logaUe)+Xf/aUO)] (99) 
t=P+i 

Under suitable regularity conditions on the weights, we can expect the consistency of 9*. 

It is well known that in the settings where the limiting distribution of a statistic is not normal, 
standard bootstrap methods are generally not consistent when used to approximate the distribution 
of the statistic. In particular when the the distribution of is very heavy-tailed in the sense that 
'^(ktl'') = for some 2 < d < 4, the Gaussian likelihood estimator is no longer assymptotically 
normal. However the least absolute deviations estimator 92 is assymptotically normal under very 
mild conditions. Hence we expect the Bootstrap methods to work under larger range of possible 
distributions for 02- 

Weighted Bootstrap for 02 As in ((99)l we define the weighted bootstrap estimators, 92 of 92 
as the minimizers of 

n 

Wnt\\og{X^ - log{at{9f)\ (100) 

t=p+i 

Let (7^ — VBWni, Wni — o'^^ {wni — 1), wlicrc Pb, Eb and Vb, respectively, denote probabilities, 
expectations and variances with respect to the distribution of the weights, conditional on the given 
data {Xi, . . . , Xn}- The following conditions on the weights are assumed: 

EBiwi) = 1 (101) 
< fc < cr2 o(n) (102) 
cin = Cov{wi,Wj) = 0{n~^) (103) 

Also assume that a'^/n decreases to as n — ^ oo. Further assume that the conditions of Result [1] hold 
with Unj = Wnj- Then it is plausible that 

\p{V^(T-\9; - §2) < . . . - P{V^{92 ~9)<x}\^0 Vx e M 
4.5 Numerical Properties 

In this section, we compare numerically the three least absolute deviation estimators with the condi- 
tional Gaussian maximum likelihood estimator for 
ARCH(l) model. Then we check the consistency of their Bootstrap analogues. 

We took the errors to have either a standard normal distribution or a standardised Student's t- 
distribution with d = 3 or d = 4 deegrees of freedom. We standardized the ^-distributions to ensure 
that their first two moments are, respectively, and 1. We took cq — 1 and ci = 0.5 in the models. 
Setting the sample size n — 100, we drew 200 samples for each setting. We used different algorithms 
to find estimates for different estimation procedures. Since the values of the parameters cq and ci 
estimated by the least absolute deviations methods differ from the numerical values specified above 
by a common factor (namely the median of the square of the distribution of et), for a given sample, 
we define the absolute error as — where cq and ci are the respective sample estimates. We 
average the error over all our samples to obtain the sample average absolute error for an estimation 
procedure. 

The table below displays the average absolute error for the different estimation procedures. The first 
column indicates distribution of et, the second column are the estimation procedures, and in the third 
column are the corresponding average error values. 
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Normal 


O3 


16.559 


t-3 


dml 


11.097 


t-3 


9i 


5.750 


t-3 


O2 


2.307 


t-3 


O3 


56.259 


t-4 




13.107 


t-4 


Oi 


7.054 


t-4 


02 


4.528 


t-4 


O3 


24.253 



Figures 3a), 3b) and 3c) present the boxplots for the absolute errors with error distributions being 
normal, tz and t4 respectively. For models with heavy-tailed errors, eg e* ~ td with c? = 3, 4 the least 
absolute dciviation estimator 62 performed best. Furthermore, the gain was more pronounced when 
the tails were very heavy, eg Ct ~ ts.From the boxplot, it can be seen that, when ct ~ ^4, except 
for a few outliers, the Gaussian maximum likelihood estimator 9 mi was almost as good as 6\ and 62- 
However, when £( ^ ts, 0mi was no longer desirable. On the other hand, when the error et was normal, 
0ml was of course the best. In fact the absolute error of 0mi was larger when the tail of the error 
distribution was heavier, which reflects the fact that, heavier the tails are, slower is the convergence 
rate; sec Hall and Yao (2003). However this is not the case for the least absolute deviations estimators 
as they are more robust against heavy tails. 

Overall the numerical results suggest that we should use the least absolute deviations estimator 62 
when Ct has heavy and especially very heavy tails, eg E(|et|^) = 00, while in general the Gausian 
maximum likelihood estimator 0^1 is desirable as long as et is not very heavy-tailed. 

Next we check the consistency of the bootstrap estimators, ^J^^^ and 62 of 6mie and 02 respectively. 
We fixed a sample of size 100 from the ARCH(l) process with standard normal errors, and used 200 
simulations for four different resampling techniques: the RB ,the m-out-of-n RB and the WB. For the 
m-out-of-n RB, we took m to be 50. Comparing the values of Vn and V*, the results of the KS test 
are: 

Two-Sample Kolmogorov-Smirnov Test 

Data: Vn and V* 
Alternative hypothesis: 
cdf of Vn does not equal the cdf of V* for at least one sample point 



Estimate 


BS Technique 


KS value 


p- value 


COml 


WB 


0.095 


0.286 


('Iml 


WB 


0.110 


0.152 


COml 


RB 


0.170 


0.005 


Clml 


RB 


0.125 


0.073 


CQml 


RB(m/n) 


0.1 


0.234 


Clml 


RB(m/n) 


0.095 


0.286 


C02 


WB 


0.095 


0.286 


C12 


WB 


0.130 


0.057 



In the table above, comi and cimi denote the estimates of cq and ci respectively using the maximum 
likelihood estimation procedure, while C02 and C12 denote the corresponding estimates using the least 
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absolute deviations estimator. From the table, it can be seen that the full sample (i.e. n-out-of- 
n)bootstrap fails, while m-out-of-n RB fares better. The reason that the full-sample RB fails to be 
consistent is that it does not accurately model relationships among extreme order statistics in the 
sample; see Fan and Yao 2003. WB does reasonably well for both maximum likelihood and least 
absolute deviations estimation procedures. 



References 

[1] Bose, A. and Chatterjee, S. (2003). Generalized Bootstrap for estimators of minimizers of convex 
functions. Journal of Stat. Planning and Inference 117, 225-239. 

[2] Brockwell, J. P. and Davis, A.R. (1990) Time Series: Theory and Methods. New York: Springer. 

[3] Chatterjee, S. and Bose, A. (2004). Generalized Bootstrap for Estimating Equations. To appear 
in Ann. Statist. 

[4] Davis, R.A., Knight, K. and Liu, J. (1992). M-estimation for autoregressions with infinite vari- 
ances. Stock. Proces. Applic. 40, 145-180. 

[5] Fan, J. and Yao, Q. (2003) Nonlinear Time Series: Nonparametric and Parametric Methods . 
New York: Springer, 125-168. 

[6] Hall, P. and Heyde, C.C. (1980) Martingale Limit Theory and its Applications. New York: Aca- 
demic Press, 6-53. 

[7] Hall, P and Yao, Q. (2003). Inference in ARCH and GARCH models with heavy-tailed errors. 
Econometrica 71, 285-317. 

[8] Peng, L and Yao, Q (2003). Least absolute deviations estimations for ARCH and GARCH models. 
Preprint. 

[9] Praestgaard, J. and Wcllner, J. A. (1993). Exchangeably Weighted Bootstrap of the General 
Empirical Process. Ann. Probab. 21, 2053-2086. 



35 



Figure 1(a) 




Figure 1(b) 



Figurel: Sample density plots of Vn and V* with ctj = 1 and (t| = 2. The green line denotes density of Vn, the red 
line for density of V*. (a) is the residual bootstrap estimator, (b) 6* is the weighted bootstrap estimator. 
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Figure 2(b) 



Figure2: Sample density plots of Vn and V* with = 1 and CTj = 10. The green line denotes density of Vn, the red 
line for density of V* . (a) is the residual bootstrap estimator, (b) is the weighted bootstrap estimator. 
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Figure 3: Box plots of the absolute errors of the maximum likelihood estimates (MLE), and the three least absolute 
deviations estimates (LADE). Labels 1, 2, 3 and 4 denote respectively the MLE, LADEl - §1, LADE2 - §2 and LADE3 
- 03. (a) Error et has normal distribution, (b) Error et has ts distribution, (c) Error et has (4 distribution. 
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